We calculate the angular scale of the acoustic oscillation from the BOOMERANG and WMAP data on the cosmic microwave background (CMB) to constrain the holographic dark energy model recently proposed by Li. We find that only the phantom-like holographic dark energy survives the cosmological tests. This is, however, inconsistent with the positive energy condition implicitly assumed in constructing Li's model. Therefore the model is marginally ruled out by the present CMB data. As a supplementary check, we also calculate the suppression of the matter density fluctuation due to the late time integrated Sach-Wolfe effect by the holographic dark energy, the result is within the tolerance of the cosmic variance. Some aspect about the saturation of the cosmic holographic bound is also discussed.